We tested for legacy effects of low-N diets offered to newly emerged second-instar spruce budworm (Choristoneura fumiferana) larvae for a duration of either one or two full instars on their growth and nutritional physiology in the sixth instar. The experimental design evaluated the effects of initial diet, final diet, and sex on energy consumption, assimilation, retention, and growth rates. Legacy effects were apparent after two instars of low-N diet exposure and were manifested by elevated (≈10%) consumption rates (RCRs) coupled to elevated (≈10%) growth rates (RGRs) and elevated (≈3%) body energy densities, i.e., heightened fat deposition. However, initial dietary N levels had no legacy main effects on food assimilation efficiencies (ADs), and gross (ECI) and net (ECD) food conversion efficiencies. RCR and AD were dependent on an initial × final diet interaction (i.e., nonlinear legacy effects). RGR depended on an initial diet × sex interaction but not on an initial × final diet interaction. Therefore, the legacy effects of low-N initial diets on RGR and body energy density were simply additive to final diet effects. Final diet universally affected all indices and interacted with sex. Low-N final diets increased RCR (≈41%) and decreased AD (14–18%) but unexpectedly increased ECD (21–24%) and RGR (≈36%). Females generally had higher performance than males on the low-N diets but often only matched males on the high-N diets. Low-N initial diets extended larval development times (≈7–26%) and lowered growth rates (6–24%) to the sixth instar, depending on duration of diet exposure, but did not affect total growth achieved by the start of the sixth instar.